Attachable depth and orientation tracker device and method of depth and orientation tracking using focal plane polarization and color camera

ABSTRACT

An imager utilizes a division of focal plane polarization and color camera to measure motion, depth and orientation of objects in a scene in real-time. In various examples, structured light, polarization-controlled discrete reflectors, and/or spatially varying discrete light sources are used to provide light of controlled polarization and color from an object in a scene to a camera. The camera utilizes a pixelated optical filter with a pattern of varying polarization filters across the pixel array, and optionally an integrated color filter pattern. Light measurements are processed to determine polarization state of light received from the object, whence orientation, position, and/or other properties of the object are determined. Systems are operable with a single camera. Applications include virtual reality, gaming, robotics, autonomous vehicles, tele-surgery, industrial automation, 3-D scanning, surveillance, and remote interaction.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/405,676, entitled “DEPTH AND ORIENTATION TRACKER USING DIVISION OFFOCAL PLANE POLARIZATION AND COLOR CAMERA,” filed Oct. 7, 2016, which isincorporated by reference herein in its entirety.

BACKGROUND

For many applications, such as remote sensing, surveillance,three-dimensional (3D) scanning/imaging, industrial inspection,metrology, medical imaging, biometrics authentication, object tracking,virtual reality and augmented reality, it may be desirable to have acompact, high resolution, portable, and fast tracker to accuratelymeasure, in real time, the movement and location of an object. Such animager should consume low power and be insensitive to vibration.

Devices using a rotating color wheel are generally sensitive tovibration. Other systems employ scanning polarizers and suffer frommotion blur. Other systems utilize a microlens array and codedapertures. Such complex systems reduce the amount of light available forimage detection, and therefore reduce the signal-to-noise ratio (SNR).The final image resolution can also be reduced considerably. Thus, themeasurement of additional information using this apparatus, such aslight direction of the light field, is made at the expense of reducedSNR and image resolution. Complex systems also have increasedcomputational burden for image processing.

SUMMARY

Methods and apparatus are disclosed for tracking objects using adivision of focal plane camera having varying polarization sensitiveelements and optionally varying wavelength sensitive elements.

In certain examples, the disclosed technology is in the form of a systemhaving one or more attachable devices, one or more light sensors, andone or more computers. Each attachable device can incorporate one ormore polarization-controlled components, which can be light-emittingelements. Each light sensor can incorporate a focal plane array which ispolarization sensitive and wavelength sensitive. The computers can beconnected to the light sensor(s).

In some examples where the polarization-controlled componentsincorporate light-emissive components, one or more of the light-emissiveelements can have uniform polarization for all angles of emission, whilein other examples, light-emissive components can have polarization thatvaries over angles of emission. In further examples, the light-emittingcomponents can incorporate a spatially varying optical filter. Thelight-emitting components can emit light at one or more wavelengths,including a wavelength that is in the infrared or ultraviolet portionsof the electromagnetic spectrum.

In additional examples, the system can include a wearable item to whichone or more attachable devices are affixed. As non-limiting examples,such wearable items can include one or more of: goggles, a helmet, avisor, a headband, a glove, a vest, a jacket, a wristband, an armband,an anklet, a legging, footwear, or an adhesive tag. The system cangenerate and store a record of the orientation of attachable devicesbased on based on light received by the light sensors from theattachable devices.

In varying examples, the disclosed system can be configured for a widerange of applications. Particularly, the system can be configured toprovide one or more of: a virtual reality or augmented reality display,user input in a gaming environment, remote control of a machine orinstrument, navigation of an autonomous vehicle, control of directionalmovement of an object, surveillance information, telesurgeryvisualization, an interactive leisure or social activity, interactivesports or athletics, or interactive training.

In further examples, the disclosed technology is in the form of anapparatus for providing virtual interactivity. First and second systemsat respective locations can generate position and/or orientation of oneor more objects at their respective locations (each location can have afinite extent, such as a room, a building, a campus, a vehicle, a road,a playing field, an arena, or an open space), wherein each detectedobject is attached to one or more of the attachable devices. Theapparatus can further include first and second displays at the locationsof the first and second systems, each configured to display objects atthe other location.

In examples, this apparatus can include one, two, several, or manyadditional instances of the disclosed system configured to generateposition and/or orientation information of objects attached toattachable devices at the respective locations of the additionalsystems; the first display can further display the objects at theseadditional system locations.

In certain examples, the disclosed technology is in the form of amethod. A system as described above or elsewhere in this disclosure isprovided, comprising one or more light sensors. At the light sensors,polarization-controlled light propagating from one or more attachabledevices is detected. At one or more computers, an analysis procedure isperformed on the detected light to determine one or more properties ofthe attachable devices. In varying examples, the determined propertiescan include one or more of: position, orientation, direction of asurface normal, speed, velocity, color, reflectance, refractive index,or bidirectional reflectance distribution.

In some examples, the analysis procedure can include a physics-basedprocedure, using e.g. Fresnel equations or a Mueller matrix formalism.The method can further include generating a virtual reality or augmentedreality display based on determined position and orientation ofattachable devices.

In certain examples, the disclosed technology is in the form of a systemhaving one or more attachable devices, one or more structuredillumination sources, one or more light sensors, and one or morecomputers. Each attachable device can incorporate one or morepolarization-controlled components, which can be reflective elements.The structured illumination sources can be configured, singly orcollectively, to illuminate the one or more attachable devices withlight having polarization structure and color structure. Each lightsensor can incorporate a focal plane array which is polarizationsensitive and wavelength sensitive. The computers can be connected tothe light sensor(s). In some examples, the system can include computersconnected to and controlling the structured illumination sources.

In additional examples, the disclosed technology is in the form of alight sensor having a polarization sensitive focal plane array ofrepeating pixels, each pixel having a block of subpixels with at least afirst subpixel and a second subpixel. The first subpixel includes afirst polarization filter configured to transmit light of a first stateof polarization and to substantially block light of a second state ofpolarization orthogonal to the first state. The second subpixel includesa second polarization filter configured to transmit light of a thirdstate of polarization (different from the first state of polarization)and to substantially block light of a fourth state of polarizationorthogonal to the third state. The light sensor can include secondrepeating pixels, each second pixel having a cluster of subpixels. Twoof the cluster subpixels have respective wavelength filters configuredto selectively transmit different first and second wavelengthsrespectively; accordingly, these subpixels transmit the first and secondwavelengths respectively.

In some examples, one or more of the block subpixels can incorporate aliquid crystal polymer retarder. Some polarization filters can include awire grid polarizer, a liquid crystal polymer polarizer, or a dichroicmaterial.

In additional examples, the disclosed technology is in the form of acamera incorporating one or more of the disclosed light sensors. Infurther examples, the disclosed technology is in the form of a systemincorporating one or more disclosed cameras, one or more structuredillumination sources, and one or more computers connected to thecameras. At least one computer can be configured to apply an analysisprocedure to data obtained from the one or more cameras to determine oneor more properties of one or more test objects illuminated by thestructured illumination. In certain examples, at least one of the thestructured illumination sources provides illumination having bothpolarization structure and color structure.

In further examples, the disclosed technology includes an imager thatutilizes a division of focal plane polarization and color camera tomeasure motion, depth and orientation of objects in a scene in real-timewith frame rate limited by the acquisition speed of the focal planearray and image processing time. In some examples of the disclosedtechnology, a system includes (1) a polarization and color structuredilluminator, (2) a polarization and color camera, (3) a computercontroller connecting to the illuminator and the camera, either by wireor by wireless connection, and (4) a set of reflectors or light emittersplaced on objects that are being tracked. The system or subsystem maycontain its own power supply, such as a battery. The usage of reflectorsor emitters can improve the accuracy of the measurement. In someexamples, there may be more than one structured illuminator toilluminate the scene at different perspectives and distances. In otherexamples, there can be more than one polarization and color camera todetect the scene at different perspectives and distances. Multiplecameras and illuminators can provide stereoscopic views of the scenes,which can facilitate the 3D reconstruction. The system design can bemodified to suit the requirements of a particular application.

In some examples, a tracking system utilizes a polarization and colorsensitive camera or sensor to track the motion of an object that has anattachable device. The attachable device either emits or reflects lightof specific and different colors and/or polarizations. In some examples,the object can be illuminated with structural illuminations of varyingcolor and polarization. Real time measurement of the polarizations andcolors received from the object and the attachable device providesinformation on the motions, orientations, and locations of the objectand subzones of the object, or of multiple objects. For example, if theobject is a person that is playing basketball, one type of polarizeddevice could be attached to the person's elbow and another type ofdevice with a different polarization could be attached to the person'shand. The camera would thus be able to distinguish the motion of theelbow from the motion of the hand and track both simultaneously.

In some applications, such an imager may be mounted in a self-drivingcar or in a drone.

Innovative methods can be implemented as part of one or more computingsystems adapted to perform an innovative method, or as part ofnon-transitory computer-readable media storing computer-executableinstructions for causing a computing system to perform the innovativemethod(s). The various innovations can be used in combination orseparately. The foregoing and other objects, features, and advantages ofthe invention will become more apparent from the following detaileddescription, which proceeds with reference to the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a depth imager 100 according to thedisclosed technologies.

FIG. 2 is a schematic diagram of a depth imager 200 according to thedisclosed technologies.

FIG. 3A depicts a periodic fringe pattern with varying intensity.

FIG. 3B depicts a plot of intensity of light as a function of position.

FIG. 3C depicts a pattern with uniform intensity.

FIG. 3D depicts a plot of polarization angle of light as a function ofposition.

FIG. 4A is a schematic diagram of a focal plane array 400 made up of anarray of pixels.

FIG. 4B depicts a set of four optical filters 410 for four subpixels.

FIG. 4C depicts a set of four optical filters 420 for four subpixels.

FIG. 4D depicts a set of four optical filters 430 for four pixels orfour subpixels.

FIG. 4E depicts a set of four optical filters 440.

FIG. 4F depicts a set of four optical filters 450.

FIG. 5A depicts a set of sixteen optical filters 510 for tiling on topof an array of pixels.

FIG. 5B depicts a set of eight optical filters 520 for tiling on top ofan array of pixels.

FIG. 5C depicts a set of twenty optical filters 530 for tiling on top ofan array of pixels.

FIG. 6 is a schematic diagram of an imaging system and an object.

FIG. 7A depicts a goggle with reflectors and/or light sources, in use bya person.

FIG. 7B depicts one facet of the goggle of FIG. 7A.

FIG. 8A depicts a light source emitting light over a steradians.

FIG. 8B depicts the polarization state of an exemplary light source atdifferent angular positions.

FIG. 8C depicts the wavelength of an exemplary light source at differentangular positions.

FIG. 8D depicts the polarization state and wavelength of an exemplarylight source at different angular positions.

FIG. 8E is a map of light directions have different polarizationeccentricity and angle over a steradians.

FIG. 8F is a map of light directions have different wavelength andpolarization angle covering 2π steradians.

FIG. 9 depicts a user wearing a goggle and being imaged by a camera.Measurement of light sources on the goggle determines the location andsurface normal of the goggle.

FIG. 10 depicts a spatially varying optical filter used to implement anexemplary light source emitting light of different polarization and/ordifferent color in different directions.

FIG. 11 is a flowchart depicting reconstruction according to an example.

FIG. 12 is a flowchart depicting reconstruction according to an example.

FIG. 13 is a flowchart depicting another example method.

FIG. 14 illustrates a generalized example of a suitable computingenvironment in which described embodiments, techniques, andtechnologies, including image processing, 3-D reconstruction, andcontrol of structured illumination can be implemented.

DETAILED DESCRIPTION

This disclosure is set forth in the context of representativeembodiments that are not intended to be limiting in any way.

I. Structured Illuminator Combining Polarization and Color

A structured light source can be constructed using a projector or alaser scanner. A predetermined set of patterns, such as random orpseudorandom codifications, binary structured codifications, or greyscale n-ary codifications, can be projected onto a scene and thereflection and/or scattered light from the projection of the scenes ismeasured. A computer can be used to control the projector or laserscanner to display different patterns as a function of time. In variousexamples, the intensity, color, polarization, coherence, and/orspatial/temporal profile of the structured light source can becontrolled to maximize signal-to-noise ratio in the image acquisition.

FIG. 1 is a schematic diagram of an exemplary depth imager 100,consisting of a color and polarization camera 101, a structuredilluminator 102, and controlling electronics 103. Light rays 105 areemitted from the illuminator 102 and are reflected from an object 104and are detected by the camera 101. In some examples, camera 101comprises a focusing lens or lens group, a focal plane imaging sensorarray (such as a polarization sensitive focal plane array), a housing,and control electronics. In other embodiments, such as those describedin context of FIGS. 7-10 and 12, object 104 can be a light-emitter; insuch embodiments illuminator 102 can be omitted. In some embodiments thecontrol electronics of camera 101 can comprise a computer, or camera 101can be coupled to a computer (not shown).

FIG. 2 is a schematic diagram of a depth imager 200 according to thedisclosed technologies, consisting of two color and polarization cameras201 and 203, a structured illuminator 202 and controlling electronics204. Light rays 205 are emitted from the illuminator 202 and arereflected from an object 206 and are detected by the two camera 201 and203 at different perspectives.

The information of the patterns can be coded in colors, such as a set ofsinusoidal fringes in white, red, green or blue intensity (FIG. 3A andFIG. 3B), or in polarization states, such as a set of uniformillumination of a single color with each pixel at a differentpolarization (FIG. 3C and FIG. 3D), or in a combination of colors andpolarization states. FIG. 3A depicts a periodic fringe pattern withvarying intensity. FIG. 3B depicts a plot of intensity of light as afunction of position corresponding to the fringe pattern of FIG. 3A.FIG. 3C depicts a pattern with uniform intensity (the stipple shadingrepresents uniform gray in the drawing), which shows no apparentfringes. FIG. 3D depicts a plot of polarization angle of light as afunction of position, for an example in which the intensity pattern isshown in FIG. 3C.

Examples of polarization state include linear, circular, and ellipticalpolarization. The wavelength of the projection is not limited to visiblelight and can be ultraviolet or infrared light. For some applications,it may be desirable to use light that is not visible to the human eye. Apolarization hyperspectral image projector has been developed by BoulderNonlinear Systems, Inc. (Colorado, USA) and can be utilized as astructured light source in examples of the disclosed technologies. Insuch a projector, a spatial light modulator can be utilized to controlthe intensity of light for each color, and two variable retarders areutilized to control the polarization state of light for each color. Forexample, a three color polarization image projector will need threespatial light modulators and six variable retarders to control the colorand polarization states. In general, increasing the number of colors inthe structured illumination increases the SNR and accuracy of themeasurement.

As used in this disclosure, the term “color” refers to anelectromagnetic spectral profile. The profile can be an emissionprofile, for example from a structured light source, a discrete lightsource, or another light source. The profile can be a transmissionprofile, for example through an optical filter, a color filter, oranother optical component. The profile can be a detection profile, forexample from a light detecting component with or without a color filter.Two colors are considered different if they can be distinguished by acolor-sensitive camera used in certain examples of the disclosedtechnology. In certain examples, a color-sensitive camera candistinguish two colors that cannot be distinguished by an average humanobserver. In certain examples, a color-sensitive camera cannotdistinguish two colors that can be distinguished by an average humanobserver. Suitable colors can include visible light colors, and includemerely as examples such common colors as red, green, blue, magenta,cyan, yellow, and white. Colors need not be visible, however, and caninclude one or more colors in the infrared or ultraviolet portions ofthe electromagnetic spectrum. In analogous fashion, the term “light”refers to electromagnetic radiation at wavelengths (in air or vacuum)between about 100 nm and 10 μm, and sometimes between about 350 nm and2.2 μm. Examples based on available laser diode sources can beassociated with wavelengths of between about 800 nm and 1700 nm.Particularly, light need not be at visible wavelengths, and can beinfrared light or ultraviolet light.

II. Color and Polarization Camera

A division of focal plane camera utilizes different micro-optical filteron each pixel or subpixel of a focal plane array to detect light ofdifferent color and polarization, as shown in FIG. 4. Focal planepolarization color cameras can be divided into several classes, forexample: broadband vs. narrowband, and linear polarization vs. fullStokes polarization.

For a broadband camera, the detector is sensitive to light fromultraviolet, through visible, to near infrared spectrum, with rangesdetermined by the absorption sensitivity of the focal plane array. Incertain examples, individual subpixel photosites of a camera sensorarray can be silicon-based (e.g., a charge-coupled device (CCD) orcomplementary oxide metal semiconductor (CMOS) sensor integratedcircuit). The image acquired by a broadband camera is monochrome. For anarrowband camera, such as a camera with a Bayer color filter array, thecamera's sensor photosites are sensitive to particular or respectiveranges of wavelengths. In certain examples, the sensed wavelength rangesmay correspond to common colors such as red, green, or blue. The imageacquired by a camera with Bayer color filter is colorful and mimics theview seen by the human visual system.

For a linear polarization camera, the detector is sensitive to light oflinear polarization states. In certain examples, this is accomplished byusing a pixelated linear micro-polarizer, in which each pixel orsubpixel can have sensitivity to a corresponding linear polarizationstate. For a camera with only one orientation micropolarizer, the camerais only sensitive to that linear polarization. For a camera with amicropolarizer having pixels or subpixels of two different orientations,the camera is sensitive to both the two linear polarization states. In acommon example of a linear polarization camera, there can be linearmicropolarizers of four orientations at 0, 45, 90 or 135 degree, whichallows the measurement of the first three components of the Stokesvector, S0, S1 and S2. The image acquired by such a camera describes thelinear polarization states of a scene. The linear micropolarizer can bea wire grid polarizer, a liquid crystal polymer polarizer using dichroicdyes, or other technologies, and combinations thereof.

As will be readily apparent to one of ordinary skill in the art, apolarization filter that transmits a particular polarization state isunderstood to substantially block light of an orthogonal polarizationstate. For example, a linear polarization filter that transmits0°-oriented linearly polarized light can substantially block light of90°-oriented linear polarization. A polarization filter that transmitsright-circularly polarized light can substantially block left circularlypolarized light. Because light of one polarization state can be resolvedinto components of light of two or more different polarization states, apolarization filter for, say, 45°-oriented light can detect linearlypolarized light having 0°- or 90°-oriented linear polarization, or evenright-circular polarization, albeit with reduced sensitivity, but willsubstantially block 135°-oriented linearly polarized light, which isorthogonal to 45°. An optical component such as a polarization filter ora color filter is said to “transmit” a certain kind of light when thecertain kind of light entering one surface of the optical componentresults in substantially the same kind of light exiting another surfaceof the optical component. Light that is not transmitted is said to beblocked, and may be reflected or absorbed. If two parallelelectromagnetic waves of substantially the same frequency and havingrespective polarization states and respective electric field vectors aresuch that the vector dot product of the electric field vectors, averagedover one period, is substantially zero for any temporal phase shiftbetween the two waves, then the two polarization states are said to be“orthogonal”. In some examples, the blocked polarization state may havetransmission through the filter that is less than 0.01%, or 0.1%, or 1%,or 2%, or 5%, or 10%, or 20%, or even 30%.

For a full Stokes polarization camera, the detector is sensitive tolight of linear, circular, and elliptical polarization states. This canbe accomplished by using a pixelated linear, circular, and/or ellipticalpolarizer. A set of four elliptical polarizers can be utilized tomeasure the complete Stokes vector, S0, S1, S2 and S3.

The Stokes vector can be estimated by a minimum of four intensitymeasurements of light passing through different polarizers. Higheraccuracy can be achieved by additional measurements. In one example of adivision of focal plane polarimeter, a pixelated retarder and a linearpolarizer are placed in front of the sensor array. A minimum of fourtypes of micro-retarder are used, oriented at four different angles, tomeasure the Stokes vector locally. The Stokes vector at each pixel iscalculated by using interpolated values of intensity measurement takenat adjacent pixels. The fast-axis orientation angles of the retarder areoptimized for the number of measurements used.

The image acquired by such a camera describes the full polarizationstates of a scene. The elliptical micropolarizer can be made of a layerof linear polarizer and a layer of retarder. In some examples, thelinear polarizer can be a wire grid polarizer, a liquid crystal polymerpolarizer using dichroic dyes, or other technologies, or combinationsthereof In some examples, the retarder can incorporate a liquid crystalpolymer.

FIG. 4A is a schematic of a polarization sensitive focal plane array 400made up of an array of pixels and subpixels. Each block or cluster offour subpixels in 400 is covered by a set of four optical filters. FIG.4B depicts an exemplary set of four optical filters 410 for foursubpixels. These optical filters can be a linear polarizer at 0° 411, aright circular polarizer 412, a linear polarizer at 90° 413, and alinear polarizer at 45° 414. FIG. 4C depicts another exemplary set offour optical filters 420, which can be a linear polarizer at 0° 421, alinear polarizer at 135° 422, a linear polarizer at 90°423, and a linearpolarizer at 45° 424. FIG. 4D depicts a set of four optical filters 430,which can be a red color filter 431, a green color filter 432, a bluecolor filter 433, and a green color filter 434. In various examples, therelative angles of linear polarizer angles can vary from the nominalvalues of 45°, 90°, or 135° by as much as 0.1°, as much as 0.2°, as muchas 0.5°, as much as 1.0°, as much as 2.0°, as much as 5.0°, or as muchas 10°. FIG. 4E depicts a set of four optical filters 440, which can beachromatic elliptical polarizers. Each elliptical polarizer is made of avertical polarizer and a retarder of retardance of 132°. Retarder 441has a fast angle at +15.1°, retarder 442 has a fast angle at +51.7°,retarder 443 has a fast angle at −51.7°, and retarder 444 has a fastangle at −15.1°. FIG. 4F depicts a set of four optical filters 450,which can be a red color filter 451, a green color filter 452, a bluecolor filter 453, and a clear (sometimes dubbed white) color filter 454.In various examples, the retarder angles can vary from the nominalvalues of ±15.1°, ±51.7°, or ±132° by as much as 0.1°, as much as 0.2°,as much as 0.5°, as much as 1.0°, as much as 2.0°, as much as 5.0°, oras much as 10°.

The disclosed technologies can be implemented with all combinations ofcolor and polarization camera. FIG. 5A depicts an exemplary set ofsixteen optical filters 510 that can be tiled on top of an array ofpixels or subpixels. The optical filters are color polarizers. Referencenumeral 511 denotes a green right circular polarizer. Reference numeral512 denotes a blue right circular polarizer. Reference numeral 513denotes a red linear polarizer at 0°. Reference numeral 514 denotes aclear linear polarizer at 45°. This example can be viewed as a clusterof four blocks or pixels (respectively red, green, clear, and blue),each block or pixel being made up of four subpixels having respectivelythree linear polarization filters and one circular polarization filter.FIG. 5B depicts an exemplary set of eight optical filters 520 which canbe tiled on top of an array of pixels or subpixels. The optical filterscan be broadband or color polarizers. Reference numeral 521 denotes agreen linear polarizer at 90°. Reference numeral 522 denotes a redlinear polarizer at 90°. 523 denotes a red linear polarizer at 0°.Reference numeral 524 denotes an infrared linear polarizer at 90°. Thisexample can be viewed as two blocks of 2×2 subpixels each. Thepolarization filter pattern is the same for each block, while the colorfilter pattern is rotated clockwise 90° going from the top block to thebottom block. FIG. 5C depicts a set of twenty optical filters 530 whichcan be tiled on top of an array of pixels or subpixels. The opticalfilters can be broadband or color polarizers. Reference numeral 531denotes a red elliptical polarizer. Reference numeral 532 denotes aninfrared elliptical polarizer. Reference numeral 533 denotes a greenelliptical polarizer. Reference numeral 534 denotes a blue ellipticalpolarizer. This example can be viewed as a cluster of five blocks orpixels (respectively green, ultraviolet, red, blue, and infrared), eachof which contains four subpixels having a repeating pattern of ellipticpolarization filters. In this example, the blocks or pixels on the toprow are offset by one subpixel horizontally from the blocks or pixels onthe bottom row.

Some examples of division of focal plane color and polarization cameraare (a) a broadband linear polarization camera, (b) a narrowband red andgreen full Stokes polarization camera, and (c) an infrared full Stokespolarization camera. Other suitable cameras can be used. Someapplications may measure only linear polarization state in one color,while other applications may measure arbitrary polarization state inthree or more colors.

In an example, camera 101 may incorporate a light sensor comprisingpolarization sensitive division of focal plane array 400. Each smallsquare in FIG. 4A can be understood to be a subpixel. In the example, arepeating block of four subpixels can be covered with a corresponding of2×2 array 410 of polarization filters 411-414 as shown in FIG. 4B.Reference numeral 411 denotes a linear polarization filter for coveringone subpixel and transmitting linearly polarized light oriented at 0°(for example, with electric field aligned with the arrow of polarizationfilter 411) and substantially blocking linearly polarized light orientedat 90° (for example, with electric field aligned perpendicular to thearrow of polarization filter 411). Reference numeral 412 denotes alinear polarization filter transmitting left-circularly polarized lightand substantially blocking right-circularly polarized light, which isorthogonal to left-circularly polarized light.

In other examples, a 2×2 array of subpixels can be covered withcorresponding arrays of polarization filters according to FIG. 4C or 4E;many other combinations and arrangements or polarization filters canalso be used. Furthermore, repeating blocks of subpixels can be ofdifferent shapes and sizes, and can include two subpixels, threesubpixels, four subpixels, five subpixels, six subpixels, sevensubpixels, eight subpixels, or even more subpixels. Additionally,repeating blocks of subpixels having one pattern of polarization filterscan be interspersed with other blocks of subpixels having anotherpattern or patterns of polarization filters, or blocks of subpixelshaving no polarization filters.

In some examples, the focal plane array can further comprise repeatingarrays of color filters covering clusters of subpixels. FIG. 5B shows acluster of eight subpixels 520 having red (R), green (G), blue (B), andinfrared (IR) filters over two blocks of four subpixels having linearlypolarized polarization filters as indicated by respective arrows in thesubpixels. In this example, the subpixels of the cluster coincide withthe subpixels of the blocks, but this is not a requirement. FIGS. 5A and5C show example configurations where the color filter subpixels eachcorrespond to four polarization filter subpixels. For example, the red(R) color filter shown in FIG. 5A comprises a block of four polarizationfilter subpixels. FIG. 5C shows a different configuration in which arepeating cluster of five color filter subpixels (green, ultraviolet,and red in the top row; blue and infrared in the bottom row) overlays anarray of twenty polarization filter subpixels. In other examples, thepolarization filter subpixels can be larger than the color filtersubpixels.

III. Computer Control of Illumination and Camera

In certain examples, a computer controls the illuminator. In certainexamples, a computer (which can be the same or a different computer)controls the camera.

IV. 3D Reconstruction Algorithms

A computer can be used to calculate the structure of a scene using oneor more 3D reconstruction algorithms. The color and polarization camerameasures the color and polarized light scattered from a sceneilluminated by structured illumination. Depending on the complexity ofthe scene, the illumination can be uniform or non-uniform. In FIG. 6, animager 610 is utilized to measure the location and surface normal 604 ofa plane 603, representing part of an object in a scene. The location ofthe imager is described by Cartesian coordinate system 601. Theorientation of the surface normal 604 of the object is described bycoordinate system 602. A light ray 605 of a specific color andpolarization state is emitted from an illuminator 611 to illuminate theobject and plane 603. The ray 606 represents light reflected and/orscattered from the plane 603, and can have a different intensity andpolarization state and is measured by a color and polarization camera612, as compared to ray 605. The information regarding the change inintensity and polarization state can be determined from the lightmeasured at camera 612 and can be utilized to calculate the location andsurface normal of objects in the scene. A 3D representation of the scenecan be constructed once the depth and surface normal of the objects aredetermined.

Various computer-implemented algorithms can be used to process datareceived from a light sensor comprising disclosed polarization sensitivefocal plane arrays. One type of algorithm calculates depth informationwith a fringe analysis method to unwrap the phase of the projectedfringe. Another type of algorithm utilizes a physics based model, suchas Fresnel equations and Mueller matrix formalism, to determine thematerial properties, such as refractive index and bidirectionalreflectance distribution function, and surface normal. For certainexemplary wavelength ranges, such as far infrared, it is also possibleto calculate the surface shape by measuring the Stokes parameters anddegree of linear polarization (DOLP) of a black body.

In examples where more than one camera is used, stereo vision techniquescan recover the 3D shape by a method of triangulation. In some examples,time-of-flight measurement can be used, with temporally modulatedillumination and detection of the phase changes of the received light.In such examples, a high frame rate can be used to achieve high spatialresolution.

As will be readily understood by one of ordinary skill in the art theparticular algorithms to implement can be adapted for a particularapplication, and can be selected based on one or more of a variety offactors, including but not limited to: computing power of the availablehardware, speed of image acquisition, number of cameras used, desiredprecision in the 3D reconstruction, latency requirements of theapplication, the particular filtered optical signals detected, detailsof the structured light, types of objects in the scene, or any priorknowledge of these objects. In some examples, a simple algorithm withlow computation overhead can be used for applications requiring fastreconstruction with medium to high accuracy.

FIG. 11 is a flowchart depicting an imaging and reconstruction procedurefor an example similar to those described above in context of FIGS. 1,2, and 6. At process block 1110, structured illumination is provided bya light source similar to light sources 102, 202, or 611. At processblock 1120, light reflected or scattered from objects in a scene isdetected by a camera similar to cameras 101, 201, 203, or 612. Atprocess block 1130, optical parameters of the light (for example, Degreeof Linear Polarization, or Stokes parameters) are calculated by acomputer similar to that described in a computing environment 1400,described further below. At process block 1140, the geometric parametersare calculated by a computer similar to that described in the computingenvironment 1400. In various examples, process blocks 1130 and 1140 maybe performed on the same or a different computer. These computers can beintegrated with controllers similar to controllers 103, or 204, or canbe integrated with an imager similar to imager 610.

V. Discrete or Point-source Emitters and Reflectors

A set of one or more reflectors or emitters can drastically improve theaccuracy of the measurement. In some examples, measurement of reflectedlight from the objects in the scene, excluding the reflectors, can beused to provide starting estimates of the location, motion, and/ororientation of the objects, as described above. Measurements fromreflectors or emitters can be used to provide additional information torefine the initial estimates. Depending on the shape and size of anobject, the number of reflectors or emitters can be increased to improvethe measurement accuracy.

The disclosed technologies can be applied to tracking of moving objectsin real time, such as a person. The person can be wearing a goggle, anarticle of clothing or a glove for tracking of the head, the body, or ahand respectively. FIG. 7A depicts a goggle with reflectors and/or lightsources, in use by a person. Also shown are three axes about which thegoggles can rotate, leading to changes in orientation, which can bedetermined using the disclosed technologies. FIG. 7B depicts one facet711 of the goggle 710, which has point reflectors or light sources, 712and 713, mounted at different locations thereon.

In some examples, point reflectors can be placed at different locationsof the goggle, clothing item, or glove. In some examples, the reflectorcan be small, having a size in a range of about 1 to 10 percent, about0.1 to 1 percent, or even about 0.01 to 0.1% compared with the size ofthe goggle, clothing item, or glove to which it is attached. Motion canbe determined by measurement of the locations of the reflectors. Thepoint reflectors can reflect light of one color or polarization state,as determined by the illumination sources.

In some examples, the reflector can be illuminated by near infraredunpolarized light. The reflector can be a linear polarized reflectoralong a predefined direction. In such examples, the reflected infraredlight is polarized along this direction. Measurement of the polarizedlight direction and location of different reflectors allows thedetermination of the surface normal of the plane or other surface onwhich the reflector is located. In other examples, each reflector may bereplaced by a light source, such as a light emitting diode, withpredefined color and polarization state(s). FIG. 8A depicts a lightsource 800 emitting light 801 over 2π a steradians. In some examples,each light source emits light of same color and polarization state inall directions. In other examples, the light source 800 can emit lightof different colors and/or different polarization states in differentdirections 801. FIG. 8B depicts the polarization state of an exemplarylight source having different polarization states for different angularpositions. The light source can emit light of one color and ninedifferent polarization states in nine different directions. 811 and 812are two directions of different polarization. FIG. 8C depicts thewavelength of an exemplary light source having different colors fordifferent angular positions. In such an example, the light source canemit light of one polarization state and nine different colors in ninedifferent directions. 821 and 822 are two directions of different color.FIG. 8D depicts the polarization state and wavelength of an exemplarylight source having different combinations of color and wavelength atdifferent angular positions. In such an example, the light source canemit light of nine different combinations of colors and polarizationstates in nine different directions. One of ordinary skill in the artwill readily appreciate that the number nine has been chosen in theseexamples purely for the purpose of illustration. In varying examples,the number of distinguished directions can be 10, 100, 1,000, 10,000, oreven larger, or the number of distinguished directions can be as smallas one. In these examples, there is a one-to-one map between thedirection of the emitted light and the color and polarization state ofthe light. FIG. 8E depicts a map of light directions having differentpolarization eccentricity and angle over 2π a steradians. Each gridsquare represents a direction; the coordinates of the grid squareindicate the ellipse eccentricity and angle of polarization for lightemitted in that direction. Thus, knowing the ellipse eccentricity andangle of polarization allows unique determination of the direction oflight emission in a reference frame of the light emitter. FIG. 8Fdepicts a map of light directions having different wavelength andpolarization angle covering 2π steradians.

Each grid square represents a direction; the coordinates of the gridsquare indicate the wavelength and angle of polarization for lightemitted in that direction. Thus, knowing the wavelength and angle ofpolarization allows unique determination of the direction of lightemission in a reference frame of the light emitter.

Such a light source can be placed on a goggle. FIG. 9 depicts a user 910wearing a goggle 940 and being imaged by a camera 920. The measurementof the color and polarization state of the light 931 can determine thedirection of the detected light in the reference frame of the lightemitter or goggle 940, which is related to the surface normal 932 of theplane of the goggle where the light source is located. Measurement oflight source(s) on the goggle determines the location and surface normal932 of the goggle. As shown in this example, the disclosed technologiesallow such determinations to be made using a single camera, unlikestereoscopic systems requiring two or more cameras.

FIG. 10 depicts a spatially varying optical filter 1003 used toimplement a light source 1000 emitting light 1004 of differentpolarization states and/or different colors in different directions.Such a light source 1000 can be implemented by using a light emittingdiode 1001, a lens 1002, and a spatially varying optical filter 1003.Light rays 1004 passing through optical filter 1003 have different colorand/or polarization states, predetermined by the optical filter 1003 andthe direction of emission. Light emitted by such a light source is aform of structured light.

FIG. 12 is a flowchart showing an imaging and reconstruction procedurefor an example utilizing light emitters. At process block 1210structured light is emitted by a light emitter similar to thosedescribed above in context of FIG. 8 or 10. At process block 1220, lightfrom the light emitter is detected by a camera similar to camera 920. Atprocess block 1230, optical parameters of the light (for example, Degreeof Linear

Polarization, or Stokes parameters) are calculated by a computer similarto computing environment 1400, described further below. At process block1240, the geometric parameters are calculated by a computer similar tocomputing environment 1400. In various examples, process blocks 1230 and1240 may be performed on the same or a different computer.

FIG. 13 is a flowchart depicting another example method. At processblock 1310, one or more light sensors are provided having respectivepolarization sensitive and wavelength sensitive focal plane arrays. Atprocess block 1320, polarization-controlled light propagated from one ormore attachable devices is detected at the light sensor(s). At processblock 1330, an analysis procedure is performed at one or more computersto determine properties of the attachable devices. The properties caninclude one or more of: position, orientation, direction of a surfacenormal, speed, velocity, color, reflectance, refractive index, orbidirectional reflectance distribution.

VI. Example Applications

Imagers and systems according to the disclosed technologies can be usedin a wide variety of applications, including virtual reality, augmentedreality, gaming, industrial automation, navigation of autonomousvehicles and robots, and tele-surgery. Applications also includedirecting remote operation of a machine or instrument, for example bysensing a user's glove at a first location to operate a machine orinstrument by matching hand motions at a second location. Applicationsalso include controlling a directional flow of energy to, from, or toavoid a sensed object. Applications also include controlling directionalmovement of the sensed object in response to its position andorientation, and controlling directional movement of a second object inrelation (for example: toward, or to avoid) the sensed object.

Imagers and systems according to the disclosed technologies can bypaired so that objects sensed at a first location can be reproduced on adisplay at a second location and vice versa, thereby enabling a widerange of remote interactive applications. Such applications includegaming, healthcare, social activities, sports or athletics, andtraining. Multiple imagers and systems according to the disclosedtechnologies, up to five, ten, 100, 1,000 or even more can be similarlycombined to create large-scale virtual worlds.

VII. Additional Examples of the Disclosed Technology

The following example features can be used individually or in anycombination or subcombination with any of the examples described herein.

An example system includes one or more attachable devices having one ormore discrete polarization-controlled components, each componentincluding one or more of a light-emitting component or a reflectivecomponent. The example system further includes zero or more illuminationsources configured to illuminate the one or more attachable devices, atleast one light sensor incorporating a polarization sensitive focalplane array, and one or more computers connected to the light sensor.

An attachable device can be attached to one or more of: goggles, ahelmet, a visor, a wearable device, a glove, a jacket, footwear, or atag. An attachable device can be attached to one or more of: a vehicle,an autonomous vehicle, a civil structure, a light pole, a utility pole,a curb, a sidewalk, a building, a container, or a person. The number ofdiscrete polarization-controlled components on at least one of theattachable devices can be one or more of: exactly one, exactly two,three to four, five to eight, nine to sixteen, seventeen to thirty-two,or thirty-three to one hundred.

In embodiments with light-emitting components, the light-emittingcomponents can emit light having one or more of the followingcharacteristics: polarization that is uniform for all angles ofemission; polarization that varies across angles of emission in oneangular direction; or polarization that varies across angles of emissionin two angular directions. The variation in polarization can bevariation in angle of polarization, or variation in ellipseeccentricity; the variation in polarization can be discretely stepped inone or more direction and/or can be continuously varied in one or moredirection. The light-emitting components can emit light having colorthat is uniform for all angles of emission; color that varies acrossangles of emission in one angular direction; or color that varies acrossangles of emission in two angular directions. The color variation can bediscretely stepped in one or more directions and/or can be continuouslyvaried in one or more directions. A light-emitting component canincorporate a spatially varying optical filter.

In embodiments with reflective components, at least one attachabledevice incorporates a reflective component, and at least one reflectivecomponent can have one or more of: preferential reflectivity of onecolor; preferential reflectivity of one polarization state; reflectivityhaving controlled variation in one angular direction; or reflectivityhaving controlled variation in two angular directions. The system canfurther include one or more illumination sources. An illumination sourcecan generate light which after reflection by a polarization-controlledreflective component, can be detected by a polarization sensitive focalplane array. The illumination source can be a structured illuminationsource and can be controlled by one or more computers.

In some embodiments, the number of light sensors is exactly one. A lightsensor can be part of a camera. A light-sensor can be color-sensitive.An attachable device can incorporate a color-controlled component, whichcan be same as or different from a polarization-controlled component.

At least one of a system's computers can be configured to apply ananalysis procedure to data obtained from the one or more light sensorsto determine one or more properties of the one or more attachabledevices. In various embodiments, the system can be configured to performone or more of the following operations: generate a virtual realitydisplay based on position and/or orientation of one or more of theattachable devices; generate position and/or orientation information ofa first one of the light sensors based on observed position and/ororientation of one or more of the attachable devices as seen by thefirst light sensor; generate and store a record of the position and/ororientation of one or more of the attachable devices; or generate andstore a record of the position and/or orientation of one or more of thecomputers.

Further, the system can be used to perform any one or more of thefollowing acts: generating a virtual reality or augmented realitydisplay; providing user input in a gaming environment; directing remoteoperation of a machine or instrument; navigating an autonomous vehicle;controlling a directional movement of an object; controlling adirectional flow of energy; surveillance; or tele-surgery.

An analysis procedure can be applied to data obtained from the one ormore light sensors of a disclosed system, to determine one or moreproperties of the one or more attachable devices. The analysis procedurecan incorporate one or more of: fringe analysis, a physics-based modelusing Fresnel equations, a physics-based model using Mueller matrixformalism, analysis of degree of linear polarization using a black-bodymodel, or stereoscopic analysis. The determined properties can includeone or more of: position, orientation, direction of a surface normal,speed, velocity, color, reflectance, refractive index, or bidirectionalreflectance distribution.

An example light sensor incorporates a polarization sensitive focalplane array having first pixels of a first repeating type, each of thefirst pixels comprising a block of subpixels. A first subpixel of theblock includes a first polarization filter configured to transmit lightof a first state of polarization and to substantially block light of asecond state of polarization orthogonal to the first state, while asecond subpixel of the block includes a second polarization filterconfigured to transmit light of a third state of polarization and tosubstantially block light of a fourth state of polarization orthogonalto the third state, the first and third states of polarization beingdifferent.

The block can include a third subpixel having a third polarizationfilter configured to transmit light of a fifth state of polarization andto block light of a sixth state of polarization orthogonal to the fifthstate, the fifth state of polarization being different from both thefirst and third states. The block can include a fourth subpixel ofhaving a fourth polarization filter configured to transmit light of aseventh state of polarization and to block light of a eighth state ofpolarization orthogonal to the seventh state, the seventh state ofpolarization being different from all of the first, third, and fifthstates. The four polarization filters of a block with four subpixels canbe organized in a variety of patterns, including those depicted in FIG.4B, 4C, or 4E.

The first and third states of polarization can be linear. The fourthstate can be the same as the first state. The second state can be thesame as the third state. Similarly, the fifth and seventh states ofpolarization can be linear, and can be oriented at 45° to the firststate of polarization. The eighth state can be the same as the fifthstate. The sixth state can be the same as the seventh state.

The block can also include one or more of fifth, sixth, seventh, oreighth subpixels, each having a respective polarization filterconfigured to transmit light of a respective polarization state. Theserespective polarization states are all different from each other andalso different from the first, third, fifth, and seventh polarizationstates. Each polarization filter can also substantially block light of apolarization state orthogonal to its respective (transmitted)polarization state.

Two among the transmitted states of polarization of a block's subpixelscan be circular and opposite to each other. Two or four among thepolarization filters can be elliptical polarizers each incorporating avertical polarizer and a retarder having a retardance of about 132° anddifferent fast axes angles selected from the group consisting of about−51.7°, about −15.1°, about +15.1°, and about +51.7°. At least oneretarder can include a liquid crystal polymer. At least one polarizationfilter can include one or more of: a wire grid polarizer, a liquidcrystal polymer polarizer, or a dichroic material.

The block can be organized as a 2×N₁ array of subpixels or as a 1×N₁array of subpixels, and the blocks can be arranged so that theirrespective first subpixels form stripes.

A light sensor can also include second pixels of a second repeating typeeach pixel having a cluster of subpixels, wherein a first subpixel ofthe cluster transmits a first color and (a) can include a color filterconfigured to selectively transmit the first color, or (b) the firstcolor is white and the first subpixel is a clear subpixel, and a secondsubpixel of the cluster can include a color filter configured toselectively transmit a second color, the first and second colors beingdifferent.

The cluster can include a third subpixel incorporating a color filterconfigured to selectively transmit a third color, different from boththe first and second colors. The cluster can include a fourth subpixelhaving a color filter configured to selectively transmit a fourth color,different from all of the first, second, and third colors. The colorfilters of a cluster and the polarization filters of a block can bearranged in various patterns, such as those shown in FIG. 5A, 5B, or 5C.

The block can be organized as a 2×N₂ array of subpixels or as a 1×N₂array of subpixels, and the clusters can be arranged so that theirrespective first subpixels form stripes. A cluster subpixel canincorporate can incorporate a block, that is, a subpixel of the secondtype of pixel can incorporate a pixel of the first type. Alternatively,a block subpixel can incorporate a cluster, that is, a subpixel of thefirst type of pixel can incorporate a pixel of the second type.

Further a first subpixel of the block can transmit the first color and(a) can include a color filter configured to selectively transmit thefirst color, or (b) the first color is white and the first subpixel is aclear subpixel, and a second subpixel of the cluster can include a colorfilter configured to selectively transmit the second color, the firstand second colors being different. A third subpixel of the block canincorporate a color filter configured to selectively transmit the thirdcolor, different from both the first and second colors. A fourthsubpixel of the block can have a color filter configured to selectivelytransmit the fourth color, different from all of the first, second, andthird colors. Any one of the transmitted colors can be selected fromred, green, blue, white, yellow, magenta, cyan, infrared, andultraviolet.

The light sensor can incorporate a charge-coupled device (CCD) lightsensor or a complementary metal oxide semiconductor (CMOS) light sensor.

An example system can include a structured illumination source, one ormore cameras incorporating a disclosed light sensor, and one or morecomputers connected to the camera(s). The system can include exactly onecamera. The structured illumination source can be coupled to one or moreof the computers.

One or more of the computers can be configured to perform an analysisprocedure on data obtained from the one or more cameras of a disclosedsystem, to determine one or more properties of one or more test objectsilluminated by the structured illumination. The analysis procedure canincorporate one or more of: fringe analysis, a physics-based model usingFresnel equations, a physics-based model using Mueller matrix formalism,analysis of degree of linear polarization using a black-body model, orstereoscopic analysis. The determined properties can include one or moreof: position, orientation, direction of a surface normal, speed,velocity, color, reflectance, refractive index, or bidirectionalreflectance distribution.

An example apparatus for providing virtual interactivity can include afirst disclosed system in a first location, and a second disclosedsystem in a second location, wherein the first and second systems areconfigured to generate position and/or orientation information of one ormore objects at the first and second locations respectively, each objectattached to at least one attachable device of the corresponding system.The apparatus also includes a first display at the first locationconfigured to display objects at the second location, and a seconddisplay at the second location configured to display objects at thefirst location.

The apparatus can further include one or more additional disclosedsystems in respective locations, wherein each additional system isconfigured to generate position and/or orientation information of one ormore objects at the location of the additional system, and wherein thefirst display is configured to display objects at the location of eachadditional system. The apparatus can further include one or moreadditional displays in respective locations, each additional displaybeing configured to display objects at the first location.

The apparatus can be used to provide one or more of: interactive gaming,interactive healthcare, interactive leisure or social activity,interactive sports or athletics, or interactive training.

VIII. Example Computing Environment

FIG. 14 illustrates a generalized example of a suitable computingenvironment 1400 in which described examples, techniques, andtechnologies, including enumeration of objects in a file system or adirectory structure, can be implemented. For example, the computingenvironment 1400 can implement all of the functions described withrespect to FIGS. 1-4, as described herein.

The computing environment 1400 is not intended to suggest any limitationas to scope of use or functionality of the technology, as the technologycan be implemented in diverse general-purpose or special-purposecomputing environments. For example, the disclosed technology can beimplemented with other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based orprogrammable consumer electronics, network PCs, minicomputers, mainframecomputers, and the like. The disclosed technology can also be practicedin distributed computing environments where tasks are performed byremote processing devices that are linked through a communicationsnetwork. In a distributed computing environment, program modules can belocated in both local and remote memory storage devices.

With reference to FIG. 14, the computing environment 1400 includes atleast one central processing unit 1410 and memory 1420. In FIG. 14, thismost basic configuration 1430 is included within a dashed line. Thecentral processing unit 1410 executes computer-executable instructionsand can be a real or a virtual processor. In a multi-processing system,multiple processing units execute computer-executable instructions toincrease processing power and as such, multiple processors can berunning simultaneously. The memory 1420 can be volatile memory (e.g.,registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flashmemory, etc.), or some combination of the two. The memory 1420 storessoftware 1480, images, and video that can, for example, implement thetechnologies described herein. A computing environment can haveadditional features. For example, the computing environment 1400includes storage 1440, one or more input devices 1450, one or moreoutput devices 1460, and one or more communication connections 1470.Measurement acquisition subsystem 1425 provides interfaces to whichcameras can be connected. Controller 1415 provides interfaces to whichstructured illumination sources can be connected.

An interconnection mechanism (not shown) such as a bus, a controller, ora network, interconnects the components of the computing environment1400. Typically, operating system software (not shown) provides anoperating environment for other software executing in the computingenvironment 1400, and coordinates activities of the components of thecomputing environment 1400.

The storage 1440 can be removable or non-removable, and includesmagnetic disks, magnetic tapes or cassettes, CD-ROMs, CD-RWs, DVDs, orany other medium which can be used to store information and that can beaccessed within the computing environment 1400. The storage 1440 storesinstructions for the software 1480 and measurement data, which canimplement technologies described herein.

The input device(s) 1450 can be a touch input device, such as akeyboard, keypad, mouse, touch screen display, pen, or trackball, avoice input device, a scanning device, or another device, that providesinput to the computing environment 1400. The input device(s) 1450 canalso include interface hardware for connecting the computing environmentto control and receive data from host and client computers, storagesystems, measurement acquisition components, control excitation sources,or to display or output data processed according to methods disclosedherein, including data acquisition systems coupled to a plurality ofsensors.

For audio, the input device(s) 1450 can be a sound card or similardevice that accepts audio input in analog or digital form, or a CD-ROMreader that provides audio samples to the computing environment 1400.The output device(s) 1460 can be a display, printer, speaker, CD-writer,or another device that provides output from the computing environment1400.

The communication connection(s) 1470 enable communication over acommunication medium (e.g., a connecting network) to another computingentity. The communication medium conveys information such ascomputer-executable instructions, compressed graphics information,video, or other data in a modulated data signal.

Some examples of the disclosed methods can be performed usingcomputer-executable instructions implementing all or a portion of thedisclosed technology in a computing cloud 1490. For example, collectionof measurement data can be executed in the computing environment (e.g.,by the measurement acquisition component 1425), while analysis of themeasurement data can be performed on remote servers located in thecomputing cloud 1490.

Computer-readable media are any available media that can be accessedwithin a computing environment 1400. By way of example, and notlimitation, with the computing environment 1400, computer-readable mediainclude memory 1420 and/or storage 1440. As should be readilyunderstood, the term computer-readable storage media includes the mediafor data storage such as memory 1420 and storage 1440, and nottransmission media such as modulated data signals.

IX. General Considerations

As used in this application the singular forms “a,” “an,” and “the”include the plural forms unless the context clearly dictates otherwise.Additionally, the term “includes” means “comprises.” Further, the term“coupled” encompasses mechanical, electrical, magnetic, optical, as wellas other practical ways of coupling or linking items together, and doesnot exclude the presence of intermediate components between the coupleditems. Furthermore, as used herein, the term “and/or” means any one itemor combination of items in the phrase.

The systems, methods, and apparatus described herein should not beconstrued as being limiting in any way. Instead, this disclosure isdirected toward all novel and non-obvious features and aspects of thevarious disclosed embodiments, alone and in various combinations andsubcombinations with one another. The disclosed systems, methods, andapparatus are not limited to any specific aspect or feature orcombinations thereof, nor do the disclosed things and methods requirethat any one or more specific advantages be present or problems besolved. Furthermore, any features or aspects of the disclosedembodiments can be used in various combinations and subcombinations withone another.

Although the operations of some of the disclosed methods are describedin a particular, sequential order for convenient presentation, it shouldbe understood that this manner of description encompasses rearrangement,unless a particular ordering is required by specific language set forthbelow. For example, operations described sequentially can in some casesbe rearranged or performed concurrently. Moreover, for the sake ofsimplicity, the attached figures may not show the various ways in whichthe disclosed things and methods can be used in conjunction with otherthings and methods. Additionally, the description sometimes uses termslike “produce,” “generate,” “display,” “apply,” “estimate,” “analyze,”and “determine” to describe the disclosed methods. These terms arehigh-level abstractions of the actual operations that are performed. Theactual operations that correspond to these terms will vary depending onthe particular implementation and are readily discernible by one ofordinary skill in the art.

Theories of operation, scientific principles, or other theoreticaldescriptions presented herein in reference to the apparatus or methodsof this disclosure have been provided for the purposes of betterunderstanding and are not intended to be limiting in scope. Theapparatus and methods in the appended claims are not limited to thoseapparatus and methods that function in the manner described by suchtheories of operation.

Some of the disclosed methods can be implemented usingcomputer-executable instructions stored on one or more computer-readablemedia (e.g., non-transitory computer-readable media, such as one or moreoptical media discs, volatile memory components (such as DRAM or SRAM),or nonvolatile memory components (such as flash drives or hard drives))and executed on a computer (e.g., any commercially available computer,including smart phones or other mobile devices that include computinghardware). Any of the computer-executable instructions for implementingthe disclosed techniques, as well as any data created and used duringimplementation of the disclosed embodiments, can be stored on one ormore computer-readable media (e.g., non-transitory computer-readablemedia). The computer-executable instructions can be part of, forexample, a dedicated software application, or a software applicationthat is accessed or downloaded via a web browser or other softwareapplication (such as a remote computing application). Such software canbe executed, for example, on a single local computer (e.g., as a processexecuting on any suitable commercially available computer) or in anetwork environment (e.g., via the Internet, a wide-area network, alocal-area network, a client-server network (such as a cloud computingnetwork), or other such network) using one or more network computers.

For clarity, only certain selected aspects of the software-basedimplementations are described. Other details that are well known in theart are omitted. For example, it should be understood that the disclosedtechnology is not limited to any specific computer language or program.For instance, the disclosed technology can be implemented by softwarewritten in C, C++, Common Lisp, Dylan, Erlang, Fortran, Go, Haskell,Java, Julia, Python, Scheme, or any other suitable programming language.Likewise, the disclosed technology is not limited to any particularcomputer or type of hardware. Certain details of suitable computers andhardware are well-known and need not be set forth in detail in thisdisclosure.

Furthermore, any of the software-based embodiments (comprising, forexample, computer-executable instructions for causing a computer toperform any of the disclosed methods) can be uploaded, downloaded, orremotely accessed through a suitable communication means. Such suitablecommunication means include, for example, the Internet, the World WideWeb, an intranet, software applications, cable (including fiber opticcable), magnetic communications, electromagnetic communications(including RF, microwave, and infrared communications), electroniccommunications, or other such communication means.

In view of the many possible embodiments to which the principles of thedisclosed subject matter may be applied, it should be recognized thatthe illustrated embodiments are only preferred examples and should notbe taken as limiting the scope of the claims to those preferredexamples. Rather, the scope of the claimed subject matter is defined bythe following claims. I therefore claim as my invention all that comeswithin the scope of these claims.

I claim:
 1. A system for tracking one or more moving objects having oneor more devices attached thereto, the system comprising: the one or moredevices, each comprising one or more polarization-controlled components,each polarization-controlled component comprising one or morelight-emitting components; one or more light sensors comprising arespective polarization sensitive and wavelength sensitive focal planearray and situated to receive respective portions of light emitted bythe light-emitting components; and one or more computers coupled to thelight sensor(s) and configured to analyze wavelength and polarization ofthe portions of light received by the light sensor(s) to determineposition and orientation of the one or more devices.
 2. The system ofclaim 1, wherein the light-emitting components have polarization that isuniform for all angles of emission.
 3. The system of claim 1, whereinthe light-emitting components have polarization that varies acrossangles of emission.
 4. The system of claim 1, wherein at least onelight-emitting component comprises a spatially varying optical filtertransmitting light of different polarization and/or different color indifferent directions.
 5. The system of claim 1, further comprising themoving objects to which the one or more devices are attached, whereinthe moving objects are one or more of: goggles, a helmet, a visor, aheadband, a glove, a vest, a jacket, a wristband, an armband, an anklet,a legging, footwear, or an adhesive tag.
 6. The system of claim 1,wherein the system is configured to provide user input in a gamingenvironment.
 7. The system of claim 1, wherein the system is configuredto generate and store a record of the orientation of the devices.
 8. Thesystem of claim 1, wherein the light-emitting components emit light atone or more wavelengths including a first wavelength that is detected bythe light sensors, and wherein the first wavelength is in an infraredspectral range or an ultraviolet spectral range.
 9. An apparatus forproviding virtual interactivity, comprising: a first system according toclaim 1 in a first location; a second system according to claim 1 in asecond location, wherein the first and second systems are configured togenerate position and orientation information of the respective one ormore objects at the first and second location, based on the determinedposition and orientation of the respective one or more devices; a firstdisplay at the first location configured to display objects at thesecond location; and a second display at the second location configuredto display objects at the first location.
 10. The apparatus of claim 9,further comprising: one or more additional systems according to claim 1in respective locations; wherein each additional system is configured togenerate position and orientation information of the respective one ormore objects at the location of the additional system; and wherein thefirst display is configured to display the respective one or moreobjects at the location of each additional system.
 11. A method,comprising: providing a system according to claim 1; at the lightsensor(s), detecting polarization-controlled light propagated from theone or more devices; and at one or more computers, performing ananalysis procedure to determine one or more additional properties of theone or more devices; wherein the additional properties are one or moreof: direction of a surface normal, speed, velocity, color, reflectance,refractive index, or bidirectional reflectance distribution.
 12. Themethod of claim 11, wherein the analysis procedure comprises aphysics-based procedure using Fresnel equations or a physics-basedprocedure using Mueller matrix formalism.
 13. The system of claim 1,wherein the polarization sensitive and wavelength sensitive focal planearray comprises first pixels of a first repeating type and second pixelsof a second repeating type, wherein: each of the first pixels comprisesa block of subpixels, wherein: a first subpixel of the block comprises afirst polarization filter configured to transmit light of a first stateof polarization and to substantially block light of a second state ofpolarization orthogonal to the first state, a second subpixel of theblock comprises a second polarization filter configured to transmitlight of a third state of polarization and to substantially block lightof a fourth state of polarization orthogonal to the third state, and thefirst and third states of polarization are different; and each of thesecond pixels comprises a cluster of subpixels, wherein: a firstsubpixel of the cluster transmits a first wavelength and comprises awavelength filter configured to selectively transmit the firstwavelength, a second subpixel of the cluster comprises a wavelengthfilter configured to selectively transmit a second wavelength, and thefirst and second wavelengths are different.
 14. The light sensor ofclaim 13, wherein at least one of the subpixels of the block comprises aliquid crystal polymer retarder.
 15. The light sensor of claim 13,wherein at least one of the polarization filters comprises one or moreof: a wire grid polarizer, a liquid crystal polymer polarizer, or adichroic material.
 16. The system of claim 1, wherein at least one ofthe polarization-controlled components has a wavelength of emission thatvaries across angles of emission.
 17. The system of claim 1, wherein atleast one of the polarization-controlled components has a wavelength ofemission that is uniform for all angles of emission.
 18. The system ofclaim 1, wherein the polarization sensitive and wavelength sensitivefocal plane array is configured to discriminate two or more states ofelliptical or circular polarization.
 19. The system of claim 1, whereinthe system is configured to control directional movement of a givenobject.
 20. The system of claim 1, wherein the system is configured toprovide a virtual reality display.
 21. The system of claim 1, whereintwo or more of the devices are attached to a same one of the movingobjects.
 22. The system of claim 1, wherein the light sensors compriseat least two distinct light sensors both receiving the respectiveportions of light emitted by the one or more light-emitting componentsof a given one of the devices.
 23. A method comprising: emittingpolarization-controlled light from one or more movable devices;receiving respective portions of the emitted polarization-controlledlight at one or more polarization-sensitive and wavelength-sensitivefocal plane arrays; and at one or more computers: analyzing wavelengthand polarization of the received light to determine position andorientation of the movable devices; and generating a virtual reality oraugmented reality display based on the determined position andorientation of the movable devices.
 24. A system for tracking one ormore moving objects having one or more devices attached thereto, thesystem comprising: the one or more devices, each comprising one or morepolarization-controlled components, each polarization-controlledcomponent comprising one or more light-emitting components, at least oneof the light-emitting components transmitting light with polarizationvarying along a first angular coordinate and color varying along asecond angular coordinate different from the first angular coordinate;one or more light sensors comprising a respective polarization sensitiveand wavelength sensitive focal plane array and situated to receiverespective portions of light emitted by the light-emitting components;and one or more computers coupled to the light sensor(s) and configuredto analyze wavelength and polarization of the portions of light receivedby the light sensor(s) to determine position and orientation of the oneor more devices.